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ABSTRACT 


Existing personalized learning systems (PLSs) have primar- 
ily focused on providing learning analytics using data from 
learners. In this paper, we extend the capability of current 
PLSs by incorporating data from instructors. We propose a 
latent factor model that analyzes instructors’ preferences in 
explicitly excluding particular questions from learners’ as- 
signments in a particular subject domain. We formulate the 
problem of predicting instructors’ question exclusion pref- 
erences as a matrix factorization problem, and incorporate 
expert-labeled Bloom’s Taxonomy tags on each question as 
a factor in our statistical model to improve model inter- 
pretability. Experimental results on a real-world educational 
dataset demonstrate that the proposed model achieves supe- 
rior prediction performance compared to several other base- 
line methods commonly used in recommender systems. Ad- 
ditionally, by explicitly incorporating Bloom’s Taxonomy, 
the model provides meaningful interpretations that help un- 
derstand why instructors exclude certain questions. Since 
instructor preference data contains their insights after years 
of teaching experience, our proposed model has the poten- 
tial to further improve the question recommendations that 
PLSs make for learners. 


Keywords 
personalized learning, educational data mining, latent factor 
model, Bloom’s Taxonomy 


1. INTRODUCTION 


Today’s education system has largely remained a “one-size- 
fits-all” learning experience in which the instructor selects a 
single learning action for all learners, ignoring their diverse 
backgrounds, interests, and goals. Modern machine learn- 
ing (ML) techniques have led to a great acceleration in the 
development of personalized learning systems (PLSs) that 
have the potential to revolutionize education by delivering a 
high-quality and affordable personalized learning experience 
at large scale. 


Current PLSs generally perform learning analytics using 
only learner data, overlooking data that instructors gener- 
ate. However, when instructors are present in educational 
settings such as traditional classrooms, they generate im- 
portant data that reveals how they prefer to interact with 
learning resources. Augmenting current learning analyt- 
ics approaches by modeling instructors’ preferences clearly 


provide advantages, since their preferences reflect years of 
teaching experience and thus provide valuable insights on 
how to utilize learning resources effectively. As a result, 
PLSs can refine their learning resource recommendations 
for learners using both learner data and these valuable in- 
sights. Additionally, analysis of instructor preferences for 
learning resources can serve as a starting point of recom- 
mending learning resource to learners when learner data is 
scarce such as at the beginning of a semester. 


In this work, we focus on a specific instance of instructors’ 
content! preferences. We collect instructors’ preferences to 
exclude questions from being given to learners in their class 
via OpenStax Tutor[13], a personalized learning and teach- 
ing platform. OpenStax Tutor has a functionality to auto- 
matically select homework assignment questions for learners 
from a question corpus. At the same time, it allows instruc- 
tors to exclude questions they do not want OpenStax Tutor 
to assign to learners in their classes from the corpus. While 
this exclusion option allows more flexibility for instructors 
to control homework assignment questions that learners re- 
ceive, manually selecting questions to exclude from a (po- 
tentially huge) corpus is a labor-intensive process. As a re- 
sult, analyzing instructors’ question exclusion behavior has 
immediate utility in automating the question exclusion pro- 
cess. 


1.1. Contributions 


With the objective of analyzing instructors’ preferences on 
assigning questions to learners on the OpenStax Tutor plat- 
form, we develop a novel latent factor model that predicts 
instructors’ question preferences in a particular subject do- 
main given previous records of whether instructors choose 
to exclude certain questions from homework assignments. 
The latent factor modeling approach is primarily inspired 
by SPARFA [10] which is a successful latent factor model 
for learner and content analysis. But more importantly, this 
approach allows flexible incorporation of prior knowledge in 
the form of meta-data into the model. Consequently, the 
model that we develop in this work can be easily extended 
to include additional information in the form of latent fac- 
tors to explain instructors’ question exclusion preferences, 


‘From now on, we will use the phrase “learning resources” 
and the word “content” interchangeably. 


Proceedings of the 10th International Conference on Educational Data Mining 290 


as well as be used in other educational data mining tasks 
where auxiliary information is available. Additionally, our 
proposed model incorporates expert-labeled Bloom’s Taxon- 
omy tags for each question to explain instructors’ question 
exclusion preferences, based on the conjecture that instruc- 
tors have varying inclinations towards different Bloom’s Tax- 
onomy tags”. 


Experimental results on a real-world educational dataset 
show that, compared to standard methods used in recom- 
mender systems, our model achieves higher overall accuracy 
in predicting instructors’ question preferences. Additionally, 
we demonstrate that our model is highly interpretable in 
that the Bloom’s Taxonomy explains question preferences of 
individual instructors, and reveals question preference pat- 
terns among instructors. Our analysis of the instructors’ 
question exclusion preferences enables PLSs to incorporate 
instructors’ insights on questions and potentially improve 
the quality of their personalized question recommendations. 


We emphasize that our proposed model is not limited to 
analyzing instructors’ question exclusion preferences; it can 
be easily modified to analyze instructors’ preferences on a 
broader range of learning resources. Therefore, our work 
serves as an initial investigation into extending the capabil- 
ity of existing PLSs with the analysis of instructor learning 
resource interaction data. 


1.2. Related Work 


We formulate the problem of predicting instructors’ question 
preferences as a matrix factorization problem underlying a 
recommender system. Recommender systems often rely on 
collaborative filtering (CF); the two most successful family 
of CF approaches to date are neighborhood-based methods 
and latent factor methods [4]. Neighborhood-based methods 
predict preferences based on neighbors chosen by some simi- 
larity measure. Latent factor methods, in particular, can be 
readily applied to education applications, resulting in ten- 
sor factorization for student modeling [15] and probabilistic 
models such as SPARFA [10], a primary source of inspiration 
for this work. However, these approaches, in their original 
form, do not have mechanisms to incorporate meta-data on 
learners and questions. Therefore, the explanatory power of 
these methods is usually limited. Our proposed model, on 
the other hand, extends the original latent factor model to 
explicitly include the Bloom’s Taxonomy tag of each ques- 
tion as meta-data, providing additional interpretability and, 
at the same time, improves prediction accuracy. 


Works including [6] and [12] incorporate external factors 
such as movie genres to improve users’ movie rating pre- 
diction in the Netflix challenge [2], but their methods do 
not directly apply to education scenarios. 


The work in [14] broadly describes a Bayesian approach to 
model instructors. While our work pursues a similar ob- 
jective, we propose a concrete model with evaluations on 


?Bloom’s Taxonomy hierarchically describes questions in 
terms of one of the six cognitive processes, including re- 
membering, understanding, applying, analyzing, evaluating, 
and creating, in increasing cognitive complexity [9]. It de- 
scribes the cognitive processes by which learners encounter 
and work with knowledge [1]. 


a real-world dataset instead of a high-level overview. [11] 
uses the k-means clustering algorithm to recommend learn- 
ing resources for instructors based on similar teaching styles 
among instructors. In addition to studying question type 
preferences, we approach the problem with a latent factor 
model instead of k-means clustering, yielding results that 
are more interpretable. 


The work in [16] compares several models in predicting learn- 
ers’ next-term grades using various features including in- 
structors’ job title, rank, and tenure status. Our work, on 
the contrary, uses data that contains instructors’ direct in- 
teraction with learning resources rather than simple demo- 
graphic information. 


2. LATENT FACTOR MODEL 

Let N, Q, K denote the total number of instructors, the 
total number of questions, and the total number of distinct 
Bloom’s Taxonomy tags, respectively. Let Y be the binary- 
valued matrix of dimension N by Q that represents instruc- 
tors’ preference for a particular course, where Yj; = 1 in- 
dicates instructor i explicitly denotes preference to exclude 
question j, and Yi; = 0 indicates no preference. Also let 
a; be a vector of dimension K that represents the question— 
Bloom’s Taxonomy tag association for question j, where ajx 
denotes the kth component of a;. aj, = 1 indicates an as- 
sociation of question 7 with Bloom’s Taxonomy tag k, and 
ajr = 0 indicates no association. 


With the above setup, we model Y as Bernoulli random 
variables: 


Yig ~ Ber($(piiaj +g hy), (1) 
Where the function ¢(-) is the sigmoid function: 


==) 


~ l+e-2" 

In the model, p; € R*, g; € R™, h; € R™ are model pa- 
rameters to be estimated, where M is the dimension of g; 
and h; (we select the value of M via cross validation). Intu- 
itively, the latent factor p; represents the instructor Bloom’s 
Taxonomy tag preference vector that reveals instructors’ dif- 
ferent preferences on each Bloom’s Taxonomy tag. The la- 
tent factors g; and h; model additional factors that also 
contribute to explaining the observed data matrix Y. 


To compare the significance of the factor p; against the fac- 
tors g; and hj, we use two simplified variants of the full 
model in Equation 1, namely P Model that involves only 
the factor p;, and GH Model that involves only factors gi 
and h;: 

P Model: Yi; ~ Ber(¢(p; aj) (2) 


GH Model: Yi; ~ Ber(¢(g7 h;)) (3) 


2.1 Optimization Algorithm 

We formulate the maximum-likelihood parameter estimation 
problem for the proposed model as an optimization problem. 
The optimization objective is given by 


ee f(P,G,H), 


where P = [pi,...,pw] denotes the matrix of instructor 
Bloom’s Taxonomy tag preference associations by stacking 
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the association vectors together. G and H are defined anal- 
ogously. The cost function f(P,G,H) is given by 


N @ 
f(P,G,H) = Yo los (1 + exp(— (pj aj + g/h,))) 


i=1 j=l 


2 2 2 
+5 olla + 2 Soe + 2 > sl. 
i=1 i=1 j=l 


The last three terms in the cost function are regularization 
terms added to prevent overfitting. A, y, and 7 are regular- 
ization parameters for the factors pi, gi, h;, respectively. 


N|R 


The above optimization problem is non-convex, but the sub- 
problems to optimize over each parameter while holding the 
others fixed are convex. We therefore employ block coordi- 
nate descent to efficiently find a local minima for the above 
optimization problem by iteratively updating each param- 
eter in turn. The update equations for the parameters are 
given by 


pi =p? — 8° s(p?, g2"4, ne) 


ge = ef 62 rept", 204 np) 


hy _ held _—6 (pr, grr ho!*), 


where 6 is the step size. The gradients of the cost function 
with respect to each parameter are given by 


Q 


) aj 
4 iy hj — 2 + A a 
dp: f (Pi; gi, hj) » L +e raster h;) + AD 
Q 
0 S- h; 
4 i, hj = a t a 
Bg, 1 (P » Bis by) “14 e7 (P? aj +e] hj) ae 
a ~ g 
aa i 8i,hj) = S : + nh;. 
Oh; P(Pi, Bi, hy) a ee Pa eee al 


At the beginning of optimization, we randomly initialize the 
model parameters p;, gi, h; for all 7,7. In each optimiza- 
tion iteration, we first loop over all 7’s to update all p; and 
g:i while holding all h,;’s fixed, and then loop over all j’s 
to update h,; using the newly calculated p;’s and g;’s. We 
repeat the above iterations until convergence, i.e., the differ- 
ence of the cost function between two iterations falls below 
a predefined threshold. 


Note that the inference problem for the P Model in Equa- 
tion 2 is convex, and optimization is straightforward via gra- 
dient descent. Since the GH Model in Equation 3 involves 
two sets of parameters and has a non-convex inference prob- 
lem, we employ the same block coordinate descent method 
as in the full model. 


2.2 Model Extensions 


We now enumerate possible extensions to the proposed model. 


First, we can incorporate additional prior information as la- 
tent factors in the model by simply including other modal- 
ities of meta-data as an additional inner product terms of 
two more latent factors inside the ¢(-) function. In this way, 
in each inner product term, one factor denotes the newly 


Table 1: Performance comparison between the pro- 
posed model and its variants, in terms of prediction 
accuracy (ACC) and area under operating charac- 
teristic curve (AUC). The proposed model achieves 
the best result among its two variants. The model 
involving the g; and h; factors achieves better per- 
formance than the model with the p; factor alone. 


Metrics 
Models ACC AUC 
Proposed Model 0.9033+0.0045 0.9592+0.0061 
P Model 0.8880+0.0047 0.8908+0.0064 
GH Model 0.9026+0.0048 0.9254+0.0058 


included meta-data modality, and the other characterizes 
the instructor’s exclusion preference in terms of that spe- 
cific modality of meta-data. Concretely, the extension of 
the model in Equation 1 has the following form: 


L 
Pine (o( So" ‘ em), (a 


l=1 


where we have replaced the inner product term pia; in 
Equation 1 with a sum of L inner product terms. Each 
ub and vj model instructor and question association of a 
particular modality of meta-data. Additionally, the dimen- 
sions of u! and vj can vary for different l’s depending on the 
mathematical representation of that meta-data modality. 


Next, it is easy to see that the same approach can be ap- 
plied to analyzing instructors’ preferences on other learn- 
ing resources. Although we specify in Equation 1 that Y;; 
represents instructor 7’s preference for question j, Yi; can 
naturally represent preferences to other contents types, by 
using 7 to index learning resources. Therefore, we can easily 
extend the proposed model in Equation 1 to analyze addi- 
tional instructor preference data with a different preference 
data matrix Y. 


3. EXPERIMENTS 


We now evaluate the prediction performance of the proposed 
latent factor model using a real-world educational dataset. 
We further showcase the interpretability of the model by 
visualizing the instructor Bloom’s Taxonomy tag preference 
vectors p;. 


3.1 Dataset 


We collect from OpenStax Tutor [13] 20 instructors’ pref- 
erences on all 896 questions of the textbook “Concepts of 
Biology” that these instructors use in their classes, resulting 
in a fully observed data matrix Y of dimension 20 by 896. 
About 15% of all entries in Y have a value of 1, meaning 
that an instructor explicitly indicates to exclude a question, 
and the rest 0, meaning that there is no such indication. 
We remind the reader that excluding a question in Open- 
Stax Tutor means that this question is excluded from the 
pool of questions that OpenStax Tutor selects from to as- 
sign to learners as personalized practice recommendations. 
We also collect the Bloom’s Taxonomy tag for each ques- 
tion, labeled by domain experts, as meta-data on the ques- 
tions. Since there are 6 distinct Bloom’s Taxonomy tags 
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Table 2: Performance comparison between the proposed model and existing collaborative filtering methods 
in terms of the four metrics. The proposed model shows superior prediction performance compared to the 


other methods on all metrics. 


Models/Methods 
Metric Full Model UBCF IBCF FSVD 
ACC 0.9033+0.0045 0.8961+0.0048 0.8895+0.0048 0.8896+0.0045 
F-1 0.6483+0.0128 0.6007+0.0158 0.5696+0.0137 0.6185+0.0158 
Precision 0.7163+0.0222 0.7070+0.0214 0.6928+0.0254 0.6964+0.0236 
Recall 0.6153+0.0227 0.5226+0.0190 0.4954+0.0159 0.5661+0.0248 


Table 3: Comparison between p;x (second row for each instructor) and the percentage of questions they 
actually excluded under each Bloom’s taxonomy tag k (first row for each instructor), for selected instructors. 
The values of p; estimated by the proposed model closely resemble the actual number of questions each 


instructor excluded. 


Bloom’s Taxonomy tag 


Instructor k=1 k=2 k=3 k=4 k=5 k=6 
5 =3 0.9% 1.6% 0.5% 1.8% 0.0% 0.0% 
= 0.058 0.083 0.038 0.216 0.075 0.084 
j=5 16.9% 16.3% 19.0% 5.5% 21.1% 33.3% 
= 0.441 0.448 0.501 0.360 1.000 0.858 
i=9 63.1% 67.8% 72.4% 67.3% 42.1% 33.3% 
a 0.826 1.000 0.985 0.924 0.583 0.215 


in total, the dimension of the question—Bloom’s Taxonomy 
tag association vector a; is kK = 6. The entries of a; cor- 
respond to Bloom’s Taxonomy tags in increasing levels of 
cognitive complexity, i.e., k = 1 represents “remembering”, 
k = 2 represents “understanding”, etc. Additionally, each 
question is only associated with one Bloom’s Taxonomy in 
our dataset. Therefore, the values of a; satisfy ajx € {0,1} 
and >>, @jx = 1 for all 7. 


3.2 Experimental Setup 

We compare our model and its variants against three 
methods frequently used in recommender systems: user- 
based collaborative filtering (UBCF), item-based collabora- 
tive filtering (IBCF), and funk singular value decomposition 
(FSVD). UBCF and IBCF use similarities among users (in- 
structors) and items (questions), respectively, and predict 
a user’s preference on an item based on the preferences of 
most similar users or items. FSVD makes the observation 
that the actual number of user and item types is much lower 
than the number of users and items, and therefore utilizes 
a low-rank model to model user—item interactions [4, 5]. [7] 
explain the detailed implementations and evaluation meth- 
ods for UCBF, ICBF, and FSVD that we use in this paper. 


We use a total of five metrics for model evaluation: (i) pre- 
diction accuracy (ACC), (ii) precision, (iii) recall, (iv) F-1 
score, and (v) area under the receiver operating characteris- 
tic curve (AUC) of the resulting binary classifier [8]. Formu- 
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las for calculating metrics (i) through (iv) are shown below: 


= ___TP+TN 
ACC ~~ TP+FP+TN+FN 
aj TP 
precision = TPLFP 
_ TP 
recall = ap EN 
F-1 —=2~x precision X recall 


precision-+recall? 


where TP denotes true positive, TN denotes true negative, 
FP denotes false positive, and FP denotes false negative. In 
the context of this paper, we treat preference for excluding 
a question, corresponding to Y;; = 1, as the positive class. 
True positive means predicting the positive class when the 
ground truth is also positive. False positive means predict- 
ing the positive class when ground truth is negative, and 
the rest follows. All metrics take on values in [0,1], with 
larger values indicating better prediction performance. We 
perform two sets of comparisons, one between the full model 
and its two variants (the P and GH models) evaluated on 
the ACC and AUC metrics, and the other one between the 
full model and UCBF, IBCF, and FSVD using ACC, F-1, 
precision, and recall. Since the AUC metric is only appro- 
priate for evaluating algorithms using probabilistic models, 
we do not evaluate the three CF methods that do not have 
an underlying probabilistic model. 


We perform 5-fold cross validation for model selection, i.e. 
choosing the best set of parameters for each model, and 
model assessment, i.e. evaluating the best model on the test 
set, according to the train-validation-test split paradigm. 
First, we randomly select 20% of all observed data and set it 
aside as test set. We then randomly partition the remaining 
80% of all data into four roughly equal-sized parts, fit the 
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Figure 1: 2D projection of instructor Bloom’s Tax- 
onomy tag preference vectors using multidimen- 
sional scaling and clustering using k-means that 
shows instructors’ diverse question exclusion pref- 
erences. Notice that instructors 3, 5, and 9 that 
we show to have very different question exclusion 
preferences also appear far apart in the plot. 


model to first three of the four parts, and validate the fitted 
model using the fourth part of the data to select the values 
of the regularization parameters using grid-search. Finally, 
we select the best performing model, fit it on all data except 
for the test set, and evaluate its performance on the test set. 
We perform 20 random partitions of the data, average the 
evaluation results, and compare the best evaluation results 
of each method. 


3.3 Results And Discussions 

Table 3 shows results for the full model, UBCF, IBCF, and 
FSVD evaluated on the ACC, F-1, Precision, and Recall 
metrics. The relatively lower Recall scores of the full model 
compared to its ACC suggests that the proposed model still 
exhibits some albeit less tendency to avoid assigning an ex- 
clusion preference label than other methods. Nevertheless, 
comparing across columns, we see that the performance of 
the full model, regardless of the choice of metric, is signifi- 
cantly better than the rest of the models, showing promise 
for the proposed latent factor model in predicting instruc- 
tors’ question exclusion preferences. 


Table 1 shows prediction performance results for the full 
model and its two variants evaluated on the ACC and AUC 
metrics. From the table, we observe that the full model 
achieves the best performance on both metrics. Further in- 
spection of the results of the two variants reveals that the 
GH Model, which involves factors g; and h;, achieves better 
results for both metrics than the P Model, which involves 
only factor p;. This implies that besides Bloom’s Taxon- 
omy, additional factors are needed in the latent factor model 
to better characterize instructors’ question exclusion pref- 
erences. Even though Bloom’s Taxonomy contribute only 
moderately to the prediction performance, the purpose of 
explicitly incorporating Bloom’s Taxonomy, as stated ear- 


Cluster Centers 


w 


Bloom's Tag 


Figure 2: Heatmap visualization of the cluster cen- 
ters that shows the radically different question ex- 
clusion preferences of each cluster of instructors. 


lier, is the power of interpretability it brings to the proposed 
model, which we demonstrate below. 


First, we use the instructor Bloom’s Taxonomy tag associ- 
ation vectors to interpret how instructors prefer to exclude 
certain questions in terms of Bloom’s Taxonomy. Table 3 
presents a comparison between the numerical values of en- 
tries in the instructor Bloom’s Taxonomy tag preference vec- 
tor p; and the percentage of questions that the correspond- 
ing instructor excludes with each Bloom’s Taxonomy tag, 
for a selected subset of instructors 7 € {3,5,9}. Comparing 
the values in the two rows for each instructor 7 in the table, 
we observe that higher values of p;;, correspond to a higher 
percentage of the questions of Bloom’s Taxonomy tag k that 
the instructor excludes. Therefore, pix reflects the degree to 
which instructor i prefers to exclude questions with Bloom’s 
Taxonomy tag k. For example, we observe from the second 
row of instructor 5 that values of pix are high for k = 5 
and k = 6, indicating that this instructor strongly prefers 
to exclude questions that involve more complex cognitive 
processes such as evaluating and creating. Second, the in- 
structor Bloom’s Taxonomy tag preference vectors uncover 
differences and patterns in instructors’ Bloom’s Taxonomy 
tag preferences. Comparing the second row of all instructors 
in Table 3, we see distinct preferences for different instruc- 
tors. For example, values of p;, for instructor 9 are high for 
k = 1,2,3,4, indicating that this instructor strongly prefers 
to not assign questions that involve simpler cognitive pro- 
cesses such as remembering, understanding, applying and 
analyzing. Such preferences are opposite to those for in- 
structor 5. Moreover, instructor 3 exhibits no obvious ex- 
clusion preference for any Bloom’s Taxonomy tags by noting 
the small values of p;, for i = 3, setting this instructor apart 
from both instructors 5 and 9. 


We further visualize patterns in instructors’ question prefer- 
ences after projecting each p; onto a 2-dimensional plane us- 
ing multidimensional scaling [3]. We then run the K-means 
algorithm to group the instructors into 3 clusters. Figure 1 
plots each p; as a point in the 2-dimensional space, where 
the color of the point denotes the cluster that the point 
belongs to. The figure shows obvious clustering patterns, 
which means that instructors exhibit only a few patterns on 
their Bloom’s Taxonomy tag preferences. Note that instruc- 
tors 3, 5 and 9 are far apart in the figure and belong to differ- 
ent clusters. Figure 2 presents a heatmap visualization of the 
cluster centers that shows distinct Bloom’s Taxonomy pref- 
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erences across the three instructor clusters. For example, the 
first and third clusters demonstrate almost entirely opposite 
Bloom’s Taxonomy preferences, where the first cluster tends 
to exclude questions with more complex cognitive process, 
whereas the third cluster tends to exclude questions with 
simpler cognitive processes. On the other hand, the second 
cluster does not exhibit strong exclusion preferences for any 
particular Bloom’s Taxonomy tag. Such clustering could 
help a PLS to recommend questions to an instructor that 
they might want to exclude, based on instructors that have 
demonstrated similar Bloom’s Taxonomy preferences. 


4. CONCLUSIONS AND FUTURE WORK 


We have presented a latent factor model that predicts in- 
structors’ question preferences, and explicitly incorporates 
questions’ Bloom’s Taxonomy tags to improve model inter- 
pretability. Evaluated on a real-world educational dataset, 
our proposed model shows superior prediction performance 
over popular collaborative filtering methods frequently used 
in recommender systems. Additionally, we demonstrated 
model interpretability by showing that the Bloom’s Taxon- 
omy captures each instructor’s question preferences reason- 
ably well, and also visualized different Bloom’s Taxonomy 
preference patterns across instructors. These encouraging 
results show the promise of using latent factor approach for 
instructors’ content preferences modeling to 1) potentially 
automate the question exclusion process in OpenStax Tutor, 
and 2) more broadly, to improve various aspects of personal- 
ized learning systems such as intelligent content recommen- 
dation that takes into account of instructors’ preferences. 


To achieve these goals, the following avenues of future re- 
search seem appropriate. First, we used only one source 
of meta-data, i.e., Bloom’s Taxonomy tags, in the proposed 
model. We have shown that the proposed model is easily ex- 
tendable to accommodate additional meta-data; moreover, 
the performance comparison between the P Model and the 
GH Model shows the need to incorporate additional factors. 
Therefore, we plan to extend the proposed model to include 
other sources of meta-data, such as the textbook chapter 
or section that each question belongs to, to improve both 
prediction accuracy and model interpretability. Second, we 
focused on instructors’ preferences in a very specific content, 
i.e., question exclusion. We are interested to see how well the 
proposed modeling approach can be adapted to analyze in- 
structors’ preference for other learning resources. Third, we 
also plan to expand our experiments from a single textbook 
to multiple textbooks and domains, in order to validate the 
proposed approach for analyzing instructor preferences on a 
wide range of contents and across different subject domains. 
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